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INTRODUCTION: 

The  purpose  of  Sound  Shore  Medical  Center’s  (SSMC)  study,  entitled  “Health  Technology 
Integration  for  Clinical,  Patient  Records  and  Financial  Management  Related  to  the  Military”,  is 
to  determine  the  relative  effectiveness  of  different  staff  training  approaches  for  nurses  regarding 
the  usage  of  a  newly-implemented  electronic  health  records  (EHR)  system  at  a  large  hospital. 
Specifically,  the  study  tested  the  effectiveness  of  two  different  training  programs  for  nurses  in 
tenns  of  developing  knowledge  of  the  EHR  system’s  procedures  and  eventual  successful 
utilization  of  the  system.  We  also  examined  the  number  of  total  hours  of  training  and  technical 
support  needed  to  achieve  proficiency  with  the  use  of  the  system,  and  the  relationship  between 
staff  characteristics  and  the  amount  of  follow-up  support  required  to  achieve  proficiency.  Prior 
studies  have  shown  that  information  technology  such  as  computerized  provider  order  entry  can 
reduce  errors  in  medication  and  treatment  and  improve  communication  between  clinicians 
(Thompson  &  Brailer,  2004;  Rind,  Safran,  &  Phillips,  1994).  While  the  use  of  EHR  systems  has 
expanded  rapidly  in  recent  years,  there  have  been  few  studies  which  looked  at  the  training  aspect 
of  EHR  implementation.  This  study  has  direct  applications  for  the  military  as  its  hospitals  seek  to 
roll  out  or  improve  their  EHR  systems,  as  well  as  for  patient  care  of  military  members  or 
dependents  who  seek  treatment  at  civilian  hospitals  due  to  travel  or  other  situations. 

BODY  OF  REPORT: 

The  accomplishments  associated  with  each  of  the  tasks  outlined  in  our  approved  Statement  of 
Work  are  as  follows: 

Installation  of  the  EHR  system :  SSMC’s  EHR  system,  which  was  under  development  for  several 
years  in  collaboration  with  a  health  information  technology  vendor,  was  rolled  out  for  employee 
use  in  October  2011. 

Development  of  Training  Schedule:  This  study  utilizes  a  randomized,  two-group  prospective 
design  in  which  two  types  of  initial  training  have  been  provided  to  our  nursing  staff.  A  list  of 
486  nurses  employed  at  both  SSMC  and  Mount  Vernon  Hospital  was  compiled  and  each  nurse 
was  randomly  assigned  (using  the  SPSS  statistical  package)  to  one  of  the  groups.  Group  training 
sessions  were  scheduled  for  each  participant  between  8/16/2011  and  9/28/2011,  with  the  16-hour 
trainings  delivered  in  two  8-hour  sessions  (two  full  days  scheduled  a  week  apart)  for  each  nurse. 
Creation/T esting  of  Online  Employee  Survey  Participating  staff  members  were  asked  to  complete 
an  online  survey  (titled  “Employee  EHR  Training  Survey”)  regarding  the  training  that  was 
created  based  on  literature  about  employee  satisfaction  and  proficiency  surveys.  Before 
administering  our  survey,  we  tested  it  with  a  group  of  “super  users”  who  were  trained  on  the 
EHR  system  prior  to  the  nurses  in  our  sample.  This  test  was  conducted  in  July  2011,  and 
included  50  respondents.  As  no  problems  or  issues  were  identified  with  the  content  of  the 
survey,  no  modifications  were  made. 

Employee  Training:  All  of  the  nurses  in  each  of  the  two  randomly-assigned  training  groups 
received  EHR  training  at  their  respective  hospital  sites  between  8/16/2011  and  9/28/2011.  Group 
1  participated  in  a  standard  instructor-led,  16-hour  classroom-based  program,  provided  in  small 
group  sessions.  Group  2  received  a  modified  training  program  consisting  of  the  same  total 
number  of  training  hours  (16),  with  two  of  these  hours  being  delivered  through  a  web-based 
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training  module.  The  total  participants  in  each  session  were  about  twenty  nurses  for  each 
training  type.  The  training  sessions  were  designed  and  delivered  by  the  hospital’s  Nurse 
Educators  using  templates  and  materials  provided  by  the  EHR  vendor.  The  topics  covered  in  the 
two-hour  web-based  module  were  the  same  as  in  the  standard  classroom  instruction  module, 
covering  creating,  viewing  and  maintaining  patient  lists;  working  with  alerts  and  flags;  entering 
and  updating  patient  infonnation;  working  with  allergies,  intolerance,  and  adverse  events;  and 
creating  clinical  summaries.  The  remaining  fourteen  hours  of  instruction  were  delivered  by 
instructors  in  the  same  manner  to  both  groups. 

Of  the  initial  roster  of  486  nurses,  a  total  of  338  nurses  received  the  sixteen-hour  EHR  training 
that  is  the  subject  of  our  study.  One  issue  that  arose  during  the  trainings  was  that  some  of  the 
nurses  in  the  original  sample  were  instead  trained  as  “super  users”  who  would  later  provide 
technical  assistance  to  other  nurses.  In  addition,  some  nurses  failed  to  attend  their  assigned 
sessions  due  to  personal  or  work-related  schedule  conflicts.  Where  possible,  these  nurses  were 
reassigned  to  other  sessions  within  their  randomly-assigned  training  groups.  However,  due  to 
scheduling  constraints  such  as  the  need  to  keep  units  adequately  staffed,  38  nurses  did  not 
receive  the  appropriate  training  for  their  assigned  group  as  a  result  of  these  absences.  These 
participants  were  therefore  dropped  from  the  analysis.  Since  the  selected  EHR  system  is  focused 
on  inpatients,  nurses  in  the  Outpatient  Department  and  Operating  Room  at  SSMC  did  not  receive 
the  full  16  hours  of  training.  These  individuals  were  also  dropped  from  the  analysis.  The 
remaining  group  of  291  nurses  (who  successfully  completed  their  assigned  training  course) 
constitutes  the  sample  that  is  used  in  our  analysis. 

Despite  the  attrition  rate  from  the  initial  sample,  we  believe  that  the  random  assignment  process 
produced  roughly  equivalent  training  groups  and  that  they  remain  roughly  equivalent.  The 
characteristics  of  the  nurses  in  each  of  the  two  training  groups  are  shown  in  the  table  below. 

Each  group  is  similar  in  terms  of  age,  experience,  and  length  of  employment  at  the  hospital,  as 
well  as  in  terms  of  education  level.  None  of  the  differences  between  the  two  groups  is 
statistically  significant  at  the  5%  level.  However,  in  one  category,  the  percentage  of  nurses  with 
a  Master’s  degree,  the  difference  is  significant  at  the  10%  level.  This  may  be  the  result  of  the 
attrition  issue  described  above  as  well  as  the  low  overall  percentage  of  participants  with  this 
level  of  education. 


Table  1:  Baseline  Characteristics  of  Randomly-Assigned  Training  Groups 
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Training  Group  1 

Training  Group  2 

N 

Mean 

N 

Mean 

Sig.  (2- 
tailed) 

(Standard 

(Standard 

Deviation) 

Deviation) 

Age 

135 

48.41 

156 

49.67 

.  965 

(10.68) 

(11.31) 

Percent 

135 

.  926 

156 

.  936 

.504 

Female 

(  .263) 

( .246) 

Percent 

135 

.  044* 

156 

.071* 

.058 

w/Master' s 
degree 

( .207) 

( .257) 

Years  of 

135 

16.24 

153 

17 . 67 

.201 

experience 

(10.16) 

(11.47) 

Years 

134 

13.03 

155 

14 . 15 

.  127 

employed 

at 

SSMC/MVH 

(10.16) 

(11 . 42) 

***Denotes  1%  significance  level;  **  denotes  5%  significance  level;  *  denotes  10%  significance  level. 


At  the  beginning  of  the  first  training  session,  all  participants  were  asked  to  complete  a  short 
survey  regarding  their  general  level  of  proficiency  with  computers  as  well  as  their  past 
experience  with  EHR  systems.  Selected  results  from  this  survey  are  presented  below.  Most 
participants  use  computers  at  home,  and  nearly  half  had  at  least  some  prior  experience  with  EHR 
systems.  The  majority  reported  fairly  frequent  (daily  or  several  times  per  week)  computer  usage, 
with  about  half  using  computers  on  a  daily  basis.  Most  participants  also  rated  their  general 
computer  skills  as  average  or  above  average,  with  only  about  7%  giving  themselves  the  lowest 
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skill  rating.  The  results  of  the  initial  proficiency  were  matched  with  measures  of  subsequent, 
post-training  proficiency  with  the  EHR  system. 

Table  2:  Initial  Proficiency  Survey  Results  (N=338) 


Question 

Yes 

No 

Have  you  ever  received  training  in  the  use  of  an  Electronic  Health 
Records  system? 

42.9% 

55.0% 

Have  you  ever  used  an  Electronic  Health  Records  system  to  record 
information  regarding  patient  care? 

46.0% 

51.9% 

Do  you  currently  use  computers  as  part  of  your  work  in  the  hospital? 

87.6% 

10.3% 

Do  you  use  a  computer  at  home? 

82.4% 

14.5% 

Do  you  have  access  to  the  internet  at  home? 

89.7% 

7.2% 

How  often  do  you  use  computers,  including  at  work  and  at  home  (please  choose  the  answer  that 
is  closest  to  the  frequency  of  your  computer  usage)? _ 


Frequency 

Percent 

Daily 

50.4% 

Several  times  per  week 

26.1% 

Once  per  week 

13.4% 

Once  per  month 

4.4% 

Never 

2.6% 

On  a  scale  of  1  to  5,  (with  1  being  not  sure  how  to  turn  a  computer  on,  and  5  being  very  good), 
how  would  you  rate  your  computer  skills? 
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Response 

Percent 

1 

7.2% 

2 

14.2% 

3 

33.1% 

4 

23.8% 

5 

18.6% 

The  Initial  Proficiency  Survey  was  completed  by  each  nurse  through  a  web-based  form  at  the 
beginning  of  their  training  session.  Each  nurse  was  asked  to  enter  their  ID  code  at  the  beginning 
of  the  survey.  As  a  result  of  mismatches  between  these  self-reported  ID  codes  and  the  codes  in 
other  datasets  containing  nurse  demographics  and  scores  on  the  proficiency  measures,  the  initial 
proficiency  measures  were  available  for  a  subset  of  237  of  the  sample.  A  comparison  of  the 
initial  proficiency  measures  of  each  training  group  is  shown  in  Table  3  below.  Independent 
samples  t-tests  showed  no  statistically  significant  differences  between  the  two  groups  on  any  of 
the  key  measures  from  the  survey. 


Table  3:  Initial  Computer  Proficiency  and  Experience  by  Training  Group 
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Training  Group  1 


Training  Group  2 


N 

Mean 

N 

Mean 

(Standard 

(Standard 

Deviation) 

Deviation) 

Percentage 
w/ prior 

116 

.43 

121 

.48 

EHR 

training 

(.50) 

(.50) 

Percentage 
w/  1  year+ 

116 

.45 

121 

.50 

EHR 

experience 

(.50) 

(.50) 

Percentage 

116 

.  91 

121 

.88 

using 
computers 
at  work 

(.28) 

(.32) 

Percentage 

116 

.86 

121 

.  82 

using 
computers 
at  home 

(.35) 

(.39) 

Percentage 
w / internet 

116 

.  92 

121 

.  92 

access  at 
home 

(.27) 

(.28) 

Percentage 

using 

116 

.55 

121 

.48 

computers 

daily 

(.50) 

(.50) 

Sig.  (2 
tailed) 


.359 


.887 


.267 


■"Denotes  1%  significance  level;  **  denotes  5%  significance  level;  *  denotes  10%  significance  level. 


Proficiency  Testing  At  the  conclusion  of  the  training  sessions,  each  participant  completed  a 
three-part  assessment  of  knowledge  of  and/or  proficiency  with  the  EHR  system.  Part  1  of  the 


post-training  assessment  consisted  of  a  20-question  written  multiple  choice  test  (developed  with 
the  assistance  of  the  vendor  providing  the  EHR  system)  designed  to  measure  knowledge  of  the 
EHR  system’s  procedures.  Part  2  of  the  post-training  assessment  involved  completing  a  series  of 
tasks  within  the  EHR  system  while  being  observed  by  an  instructor,  with  points  recorded  by  the 
instructor  for  each  task  completed  successfully.  Finally,  each  participant’s  proficiency  with  the 
actual  use  of  the  system  after  the  “go-live”  implementation  date  was  assessed  by  their  supervisor 
or  a  Nurse  Educator  using  a  27-item  observational  protocol,  with  each  item  rated  on  a  five-point 
scale  The  number  of  nurses  for  whom  a  score  was  available  on  each  measure  varies,  as  not  all 
nurses  completed  all  of  the  assessments,  and  there  were  some  issues  with  matching  ID  codes  for 
Part  1  as  this  test  was  delivered  through  a  web-based  form  in  which  nurses  entered  their  own 
(and  sometimes  incorrect)  ID  codes. 

While  these  instruments  had  not  been  validated  to  establish  the  relationship  between  test  scores 
and  proficiency  with  the  use  of  the  EHR  system,  the  items  tested  include  basic  system 
procedures  that  are  necessary  to  achieve  proficient  use  of  the  system.  We  chose  to  employ  the 
three  different  approaches  to  measuring  proficiency  -  a  written  test,  a  practical  exam,  and  an 
observational  rating  by  a  supervisor  -  in  part  due  to  the  lack  of  availability  of  existing,  validated 
instruments  in  the  hope  of  showing  consistency  between  the  measures.  Table  4  shows  the 
correlations  between  each  of  the  measures  as  well  as  between  each  measure  and  the  amount  of 
follow-up  training  needed.  Part  2  (in  which  EHR  tasks  were  completed  and  scored  by  an 
instructor)  is  most  likely  the  best  measure  of  proficiency  of  the  three  as  it  is  positively  correlated 
with  both  of  the  other  assessments,  and  negatively  correlated  with  the  number  of  minutes  of 
follow-up  required  (in  other  words,  higher  scores  were  associated  with  fewer  minutes  of  follow¬ 
up  training). 

Table  4:  Correlations  -  Proficiency  Scores  and  Minutes  of  Follow-up _ 


Part  1 

Part  2 

Supervisor 

Minutes  of 

Score 

Score 

Rating 

Follow-up 

Part  1 

Pearson  Correlation 

1 

.226** 

.127 

-.117 

Score 

Sig.  (2-tailed) 

.001 

.065 

.073 

N 

235 

230 

214 

235 

Part  2 

Pearson  Correlation 

.226** 

1 

.420** 

-.131* 

Score 

Sig.  (2-tailed) 

.001 

.000 

.029 

N 

230 

279 

252 

279 

Supervisor 

Pearson  Correlation 

.127 

.420** 

1 

-.122 

Rating 

Sig.  (2-tailed) 

.065 

.000 

.050 

N 

214 

252 

257 

257 

Minutes  of 

Pearson  Correlation 

-.117 

-.131* 

-.122 

1 

Follow-up 

Sig.  (2-tailed) 

.073 

.029 

.050 

N 

235 

279 

257 

289 

“Correlation  is  significant  at  the  0.01  level  (2-tailed). 
‘Correlation  is  significant  at  the  0.05  level  (2-tailed). 
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The  differences  in  the  proficiency  scores  between  the  training  groups  were  examined  as  part  of 
the  test  of  our  hypothesis  that  nurses  who  receive  web-based  training  as  a  substitute  for  part  of 
the  classroom-based  training  will  show  higher  initial  levels  of  knowledge  of  EHR  system 
procedures.  The  results  of  our  analysis  of  the  proficiency  test  scores  are  presented  below  in  the 
section  titled  “Results  of  Final  Data  Analysis”. 

Deployment/Administration  of  Online  Survey  The  tables  below  present  some  of  the  key  results 
from  a  post-training  anonymous  survey  of  participants  regarding  their  satisfaction  with  the 
training.  Since  this  survey  was  anonymous,  these  results  cannot  be  linked  to  the  proficiency 
scores  or  other  measures  but  do  give  us  some  insight  into  how  participants  felt  about  the 
effectiveness  of  the  training.  The  Employee  EHR  Training  Survey  was  initially  deployed  in  July 
2011  for  testing  with  an  initial  group  of  “super  users”  who  received  more  extensive  training  on 
the  EHR  system  than  the  nurses  in  our  sample.  Each  nurse  in  our  study  sample  was  then  asked  to 
complete  the  Employee  EHR  Training  Survey  at  the  end  of  their  last  training  session.  The  total 
number  of  respondents  is  higher  than  the  number  of  participants  in  our  score  analysis,  as  this 
survey  was  taken  by  trainees  who  were  subsequently  dropped  from  the  analysis  due  to  the 
reasons  outlined  previously  (because  they  were  supervisors,  missed  the  assigned  training,  etc.). 
Overall,  the  nurses  seem  to  be  fairly  positive  about  the  utility  of  the  new  system,  but  somewhat 
more  ambivalent  about  their  readiness  to  use  the  system  immediately  post-training.  Items  which 
involved  rating  statements  about  the  effectiveness  of  the  system,  such  as  whether  it  would 
improve  patient  outcomes  and  the  accuracy  of  information  tracking,  were  rated  highly  (averaging 
greater  than  4  out  of  5).  However,  only  about  half  of  respondents  felt  that  they  were  ready  to  use 
the  new  EHR  system  immediately  after  training,  and  only  about  7%  reported  being  “very 
comfortable”  with  the  system  at  the  end  of  the  training.  It  should  be  noted  that  this  survey  was 
administered  immediately  after  the  training  sessions  and  before  the  nurses  began  actually  using 
the  system,  so  these  are  initial  reactions  following  the  16-hour  training  session. 
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Table  5:  Responses  to  anonymous  post-training  satisfaction  survey 


Did  part  of  your  training  in  voive  web  based  training  ? 

Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

58.0% 

221 

No 

42.0% 

160 

Total 

381 

How  comfortable  are  you  with  the  new  system  after  the  training ? 

Answer  Options 

Response 

Percent 

Response 

Count 

Very  comfortable 

7.3% 

28 

Somewhat  comfortable 

69.0% 

263 

Not  at  all  comfortable 

23.6% 

90 

Total 

381 

/  feet  ready  to  use  the  new  Electronic  Health  Record  system. 

Answer  Options 

Response 

Percent 

Response 

Count 

Yes 

50.7% 

189 

No 

49.3% 

184 

Total 

373 
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Please  rate  how  you  feel  about  the  following  statements: 


The  old  (paper  records) 
system  is  more  efficient. 


The  new  Electronic  Health 
Record  system  will  result 
in  better  patient.. 


The  new  system  will 
make  my  work  easier 


I  received  adequate 
training  on  the  new  system 


My  manager  visibly  commits 
to  the  implementation 
of  the  newElectroni.  .. 

Information  will  be 
more  completely  recorded 
in  the  new  system 

Information  will  be 
more  accurately  recorded 
in  the  new  system. 


The  rating  scale  for  the  above  questions  is  as  follows: 

1  =“Definitely  Not” 

2  =“Maybe  Not” 

3=“Not  Sure” 

4=“Maybe” 

5=“Definitely” 

Table  6  shows  a  set  of  bivariate  correlations  between  self-reported  personal  characteristics  of 
respondents  and  their  level  of  satisfaction  with  the  EHR  training  and  perceived  readiness  to 
begin  using  the  EHR  system.  Satisfaction  with  the  training  was  measured  using  a  five  point 
rating  scale;  perceived  readiness  to  begin  using  the  system  was  measured  through  two  yes/no 
questions.  Increasing  age  was  associated  with  both  lower  levels  of  satisfaction  with  the  training 
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Table  6:  Anonymous  Survey:  Correlations  -  Nurse  Characteristics,  Satisfaction  with  Training,  Readiness 


How 

Have  you  ever 

comfortable  are 

In  your  opinion, 

used  an 

1  feel  ready  to 

you  with  the 

how  good  are 

electronic 

1  received 

use  the  new 

new  system 

your  computer 

health  records 

adequate 

Electronic 

after  the 

skills? 

system  in 

training  on  the 

Health  Record 

training? 

Age: 

Masters 

English 

(Scale,  1-5) 

another  job? 

new  system. 

system. 

(Scale,  1-3) 

Age 

Pearson  Correlation 

1 

.006 

-.161" 

-.471 " 

-.177" 

-.183" 

-.152" 

-.207** 

Sig.  (2-tailed) 

.920 

.003 

.000 

.001 

.001 

.007 

.000 

N 

338 

338 

338 

334 

334 

318 

318 

325 

Masters  or  Above 

Pearson  Correlation 

.006 

1 

.123* 

.094 

.103 

-.083 

.014 

.101 

Sig.  (2-tailed) 

.920 

.024 

.086 

.060 

.138 

.802 

.070 

N 

338 

338 

338 

334 

334 

318 

318 

325 

Native  English  Speaker 

Pearson  Correlation 

-.161** 

.123' 

1 

.178" 

.051 

-.128* 

-.015 

.039 

Sig.  (2-tailed) 

.003 

.024 

.001 

.349 

.022 

.796 

.481 

N 

338 

338 

338 

334 

334 

318 

318 

325 

In  your  opinion,  how  good 

Pearson  Correlation 

-.471" 

.094 

.178" 

1 

.322" 

.118* 

.255” 

.318** 

are  yourcomputerskills? 

Sig.  (2-tailed) 

.000 

.086 

.001 

.000 

.036 

.000 

.000 

(Scale,  1-5) 

N 

334 

334 

334 

334 

334 

318 

318 

325 

Have  you  ever  used  an 

Pearson  Correlation 

-.177" 

.103 

.051 

.322" 

1 

.052 

.291" 

.243" 

electronic  health  records 

Sig.  (2-tailed) 

.001 

.060 

.349 

.000 

.353 

.000 

.000 

system  in  another  job? 

N 

334 

334 

334 

334 

334 

318 

318 

325 

1  received  adequate  training 

Pearson  Correlation 

-.183" 

-.083 

-.128* 

.118* 

.052 

1 

.407" 

.452" 

on  the  new  system. 

Sig.  (2-tailed) 

.001 

.138 

.022 

.036 

.353 

.000 

.000 

N 

318 

318 

318 

318 

318 

318 

318 

318 

1  feel  ready  to  use  the  new 

Pearson  Correlation 

-.152" 

.014 

-.015 

.255" 

.291" 

.407" 

1 

.469" 

Electronic  Health  Record 

Sig.  (2-tailed) 

.007 

.802 

.796 

.000 

.000 

.000 

.000 

system. 

N 

318 

318 

318 

318 

318 

318 

318 

318 

How  comfortable  are  you 

Pearson  Correlation 

-.207” 

.101 

.039 

.318" 

.243" 

.452" 

469" 

1 

with  the  new  system  after 

Sig.  (2-tailed) 

.000 

.070 

.481 

.000 

.000 

.000 

.000 

the  training?  (Scale,  1-3) 

N 

325 

325 

325 

325 

325 

318 

318 

325 

"Corre  lation  is  significant  at  the  0.0 1  level  (2-tailed). 
'Correlation  is  significant  at  the  0.05  level  (2-tailed). 
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and  lower  perceived  readiness.  Higher  levels  of  self-reported  computer  skills  as  measured  by  a 
five-point  scale  were  associated  with  higher  levels  of  both  satisfaction  and  perceived  readiness. 
Previous  EHR  experience  was  not  associated  with  differences  in  satisfaction  with  the  training 
but  was  associated  with  higher  levels  of  perceived  readiness  to  use  the  system.  Education  level, 
as  measured  by  reporting  having  a  Master’s  degree  or  above,  was  not  associated  with  either 
satisfaction  or  perceived  readiness  to  use  the  system.  Being  a  native  English  speaker  was 
associated  with  lower  levels  of  satisfaction  with  the  training  but  was  not  associated  with  the 
measures  of  perceived  readiness  to  use  the  system. 

The  survey  results  also  show  variations  in  perceived  readiness  to  use  the  new  EHR  system 
between  the  group  that  received  the  web-based  training  component  and  the  group  receiving  only 
the  traditional  instructor-led  approach.  As  shown  in  Table  7,  about  55%  of  survey  respondents 
who  reported  participating  in  the  web-based  component  indicated  that  they  felt  ready  to  use  the 
new  EHR  system,  versus  41%  of  those  who  reported  participating  only  in  the  traditional  training. 
However,  both  groups  reported  similar  ratings  of  satisfaction  with  their  training  program, 
averaging  3.58  and  3.49  respectively  on  the  five-point  rating  scale.  It  should  be  noted  that  since 
this  survey  was  anonymous,  these  results  rely  on  self-reported  participation  in  each  of  the 
training  groups  as  opposed  to  the  actual  random  assignments  used  in  the  other  parts  of  the  study. 

Finally,  the  anonymous  survey  included  an  open-ended  question  regarding  how  the  training 
program  could  be  improved  (“How  could  the  training  be  improved  to  enable  you  to  learn  more 
about  the  electronic  medical  record  system?”).  The  most  common  response  by  far  was  some 
variation  on  “more  practice”  or  “more  hands-on  time”.  Among  the  188  non-“super  users”  who 
answered  this  question,  65  responses  fell  into  this  category.  For  the  same  group,  39  responded 
with  “more  training  time”  or  a  similar  phrase,  the  next  most  common  response.  Other  responses 
were  not  as  frequent  and  thus  were  not  as  easily  categorized,  with  most  being  expressed  by  only 
one  individual  or  a  handful  of  individuals.  A  few  respondents  stated  that  the  training  should  be 
“unit  specific”  as  opposed  to  mixing  nurses  from  different  units.  Some  suggested  that  the 
trainees  should  be  grouped  by  ability  level,  and  that  there  should  be  additional  trainers  in  each 
session.  Only  a  small  number  commented  on  the  web-based  component,  with  four  respondents 
suggesting  that  it  should  be  eliminated,  while  three  suggested  that  it  should  be  expanded. 


Table  7:  Perceived  Post-Training  Readiness  to  Use  New  EHR  System  by  Training  Group 
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Web-based  training 

Traditional  training 
only 

N 

Mean 

(Standard 

Deviation) 

N 

Mean 

(Standard 

Deviation) 

Sig . 

(2- 

tailed) 

I  feel  ready 
to  use  the 

new 

Electronic 
Health  Record 
System 

(Yes/No) 

189 

.55** 

(  .499) 

129 

.41** 

(.494) 

.015 

NOTE:  This  table  excludes  “super  users” 

***Dcnotes  1%  significance  level;  **  denotes  5%  significance  level;  *  denotes  10%  significance  level. 


Collection  of  Samples  of  System  Usage  Data:  Our  original  plan  called  for  collection  of  an 
additional  proficiency  measure,  based  on  randomly  sampling  five  patient  records  for  each 
participant  at  the  conclusion  of  the  training  period  and  detennining  if  a  list  of  critical  items  was 
successfully  completed  within  each  record.  Our  preliminary  discussions  with  our  EHR  vendor 
during  the  planning  of  the  project  had  led  us  to  believe  that  this  was  feasible.  However,  as  we 
moved  to  implement  this  measure  we  were  told  by  the  vendor  that  the  way  the  system  is 
structured  that  this  is  not  possible.  Data  entry  errors  are  flagged  immediately  by  the  system  and 
will  generally  not  allow  the  record  to  be  saved;  they  system  does  not  record  these  errors  and 
cannot  be  modified  by  the  vendor  to  produce  this  infonnation.  We  were  therefore  forced  to 
abandon  the  use  of  data-entry  errors  as  a  performance  measure.  As  an  alternative  measure  of 
competency,  we  developed  a  structured  assessment  based  on  a  27-item  observational  protocol 
administered  by  a  member  of  the  nursing  department,  such  as  a  nursing  director,  manager,  or 
educator.  Each  nurse  was  asked  to  demonstrate  skills  and  to  articulate  the  purpose  or  reason  for 
specific  information  that  must  be  entered  into  the  system.  The  knowledge  and  skills  that  were 
selected  for  assessment  are  necessary  to  ensure  that  a  valid  EHR  is  created  for  each  patient. 

Interim  Data  Analysis/Interim  Evaluation  Report:  The  interim  data  analysis  described  in  our 
Statement  of  Work  was  submitted  as  our  Annual  Report  in  October  2011.  The  infonnation  in  the 
interim  data  analysis  is  incorporated  in  this  report. 
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Results  of  Final  Data  Analysis :  This  section  presents  the  final  results  our  analysis  the  data 
collected  relative  to  our  hypotheses  and  specific  research  questions.  The  main  goals  of  our 
project  were  to  test  two  different  approaches  to  staff  training  for  registered  nurses  on  levels  of 
knowledge  regarding  the  usage  of  a  new  Electronic  Health  Record  (EHR)  system,  as  well  as  to 
determine  the  total  number  of  training  and  support  hours  needed  to  achieve  proficiency  with  the 
EHR  system.  Therefore,  the  hypotheses  tested  are  as  follows: 

•  Registered  Nurses  who  receive  web-based  training  in  addition  to  classroom-based 
training  will  show  higher  initial  levels  of  knowledge  of  EHR  system  procedures. 

•  Registered  Nurses  with  different  personal  characteristics  (age,  years  of  formal  education, 
native  language,  use  of  a  computer  at  home)  will  require  different  levels  of  subsequent 
training  and  technical  support  in  order  to  achieve  proficiency  and  comfort  using  the  new 
Electronic  Health  Record  system.  We  expect  users  with  higher  levels  of  formal 
education,  who  speak  English  as  a  native  language,  and/or  have  higher  prior  levels  of 
computer  proficiency  to  require  the  fewest  total  hours  of  training  and  support  to  achieve 
proficiency  with  the  use  of  the  new  EHR  system. 

Our  specific  research  questions  are  as  follows: 

1 .  Does  a  training  plan  that  includes  a  web-based  component  as  well  as  classroom-based 
training  result  in  higher  initial  levels  of  knowledge  regarding  use  of  the  EHR  system  for 
nurses  than  a  classroom-only  plan  of  the  same  duration? 

2.  How  many  hours  of  training  and  technical  support  are  required  for  nurses  to  achieve 
proficiency  in  the  use  of  the  EHR  system? 

3.  Are  personal  characteristics  (age,  job  type,  gender,  prior  computer  literacy)  related  to  the 
amount  of  training/technical  support  needed  to  achieve  proficiency  with  the  EHR 
system?  Do  these  characteristics  affect  satisfaction  with  the  new  EHR  system? 

Findings  Regarding  Web-Based  vs.  Traditional  Training  Model  on  Training  Outcomes 
The  following  tables  show  the  average  test  scores  by  training  group  for  each  of  the  three 
proficiency  measures.  Part  1  of  the  post-training  assessment  was  a  20-question  written  multiple 
choice  test.  Part  2  of  the  post-training  assessment  involved  completing  a  series  of  tasks  within 
the  EHR  system,  with  points  recorded  by  the  instructor  for  each  task  completed  successfully. 
Finally,  each  participant’s  proficiency  with  the  actual  use  of  the  system  after  the  “go-live” 
implementation  date  was  assessed  by  their  supervisor  or  a  Nurse  Educator  using  a  27-item 
observational  protocol,  with  each  item  rated  on  a  five-point  scale. 


Table  8:  Post-Training  Assessment  -  Part  1  (Maximum  possible  score  -  40  points) 
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Training  Group  1 

Training  Group  2 

N 

Mean 

N 

Mean 

Sig.  (2- 

tailed) 

(Standard 

(Standard 

Deviation) 

Deviation) 

Part  1 

108 

26.48 

127 

25. 97 

.360 

Score 

(4.33) 

(  4.22) 

Highest  score=34;  lowest  score=10. 

***Denotes  1%  significance  level;  **  denotes  5%  significance  level;  *  denotes  10%  significance  level. 


Table  9:  Post  Training  Assessment-Part  2  (Maximum  possible  score  -  60  points) 


Training  Group  1 

Training  Group  2 

N 

Mean 

N 

Mean 

Sig.  (2- 

tailed) 

(Standard 

(Standard 

Deviation) 

Deviation) 

Part  2 

137 

52 . 15 

142 

53.42 

.322 

Score 

(9.97) 

(11.44) 

Highest  score=60;  lowest  score=0. 

***Dcnotes  1%  significance  level;  **  denotes  5%  significance  level;  *  denotes  10%  significance  level. 


Table  10:  Post  Training  Assessment  -  Supervisor  Proficiency  Rating  (Maximum  possible 
score  - 135  points) 
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Training  Group  1 

Training  Group  2 

N 

Mean 

N 

Mean 

Sig.  (2- 

tailed) 

(Standard 

(Standard 

Deviation) 

Deviation) 

Supervisor 

129 

122 .16 

128 

122 . 12 

.  981 

Rating 

Score 

(13.65) 

(16.79) 

Highest  score=135;  lowest  score=54. 

***Denotes  1%  significance  level;  **  denotes  5%  significance  level;  *  denotes  10%  significance  level. 


As  the  tables  show,  the  two  groups  received  very  similar  scores  on  each  assessment.  The  number 
of  nurses  for  whom  a  score  was  available  on  each  measure  varies,  as  not  all  nurses  completed  all 
of  the  assessments.  Based  on  independent  samples  t-tests,  there  are  no  statistically  significant 
differences  between  the  groups  on  any  of  the  three  scores.  These  findings  did  not  support  our 
hypothesis  that  nurses  who  receive  web-based  training  as  a  substitute  for  part  of  the  classroom- 
based  training  will  show  higher  initial  levels  of  knowledge  of  EHR  system  procedures.  However, 
it  should  be  noted  that  this  change  in  the  training  model  produced  essentially  the  same  results  as 
the  fully  classroom-based  model,  which  may  have  implications  regarding  reducing  overall 
training  costs  as  discussed  in  the  Conclusions  section  below. 

Findings  Regarding  Level  of  Training/Technical  Support  Needed 

The  level  of  additional  training  and  technical  support  needed  by  each  nurse  was  determined  by 
recording  the  number  of  minutes  of  such  assistance  provided  to  each  individual  after  the  new 
EHR  system  was  placed  into  service.  For  a  two-week  period  following  beginning  at  the  point  at 
which  the  system  “went  live”  throughout  both  hospitals,  each  nursing  supervisor  as  well  as  a  set 
of  roving  trainers  were  instructed  to  record  the  time  spent  on  assisting  each  nurse  with  the  use  of 
the  new  system.  These  contacts  were  recorded  on  paper  rosters  containing  each  nurse’s  name, 
the  number  of  minutes  of  assistance  provided,  the  date,  and  the  nurse’s  unit  within  the  hospital. 
Administrators  later  added  identification  codes  to  each  record,  removed  the  names,  and 
forwarded  the  data  to  the  researchers. 

Table  1 1  below  summarizes  this  follow-up  assistance  data.  The  average  number  of  minutes  of 
follow-up  training  provided  to  all  members  of  the  sample  was  53  minutes;  however,  there  was  a 
great  deal  of  individual  variation  in  the  number  of  minutes  provided.  About  half  of  the 
participants  showed  no  subsequent  training  contacts,  while  a  few  had  several  hours  of  additional 
training  over  this  period. 
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Table  11:  Follow-up  Training/Technical  Assistance  Provided 

Number  of  Contacts  Recorded  I 


Average  Number  of  Minutes  per 
Contact 


27.24 


(34 . 05) 


(Standard  Deviation) 


Minimum  number  of  minutes  in  a 
single  contact 


Maximum  number  of  minutes  in  a 
single  contact 


Unique  individuals  receiving 
assistance  (Percentage  of 
sample,  N=289) 


Average  number  of  minutes  of 
assistance  per  individual  for 
entire  sample  (including  those 
receiving  no  assistance) 


Average  number  of  contacts  per 
individual  among  nurses 
receiving  assistance 


Average  total  minutes  of 
assistance  per  individual  only 
among  nurses  receiving 
assistance 


Maximum  number  of  contacts 
provided  to  an  individual 


Maximum  number  of  minutes 

550 

provided  to  an  individual 

Table  12:  Average  Number  of  Minutes  of  Follow-up  Assistance  by  Group 


Training  Group  1 

Training  Group  2 

N 

Mean 

N 

Mean 

Sig.  (2- 

tailed) 

(Standard 

(Standard 

Deviation) 

Deviation) 

Minutes 

143 

48 . 97 

146 

57.08 

.471 

of 

Follow-up 

(92 . 94) 

(97 . 84) 

***Denotes  1%  significance  level;  **  denotes  5%  significance  level;  *  denotes  10%  significance  level. 


Applying  an  independent  samples  t-test  to  the  above  data  (Table  12)  shows  no  statistically 
significant  difference  in  the  mean  number  of  minutes  of  follow-up  assistance  provided  to  each 
group.  However,  it  is  likely  that  this  standard  approach  is  not  advisable  in  this  situation  as  the 
dependent  variable  (minutes  of  assistance  provided)  is  not  normally  distributed,  thereby  violating 
one  of  the  key  assumptions  for  the  use  of  a  t-test.  The  distribution  of  the  dependent  variable  is 
shown  in  Table  13.  An  alternative  approach  to  modeling  this  type  of  count  data  -  in  which  many 
cases  have  a  value  of  zero  due  to  the  absence  of  the  event  being  counted  -  is  to  employ  Poisson 
or  negative-binomial  distributions,  which  may  provide  a  better  fit  to  the  data.  Therefore,  as  a 
further  check,  bivariate  Poisson  and  negative-binomial  regressions  were  estimated  with  training 
group  as  a  binary  predictor  and  the  number  of  minutes  of  follow-up  as  the  dependent  variable. 
The  Poisson  model  indicated  a  statistically  -significant  difference  between  the  two  groups,  but 
was  a  poor  fit  to  the  data  as  there  was  evidence  of  overdispersion.  The  negative  binomial  model 
was  indicated  to  be  a  better  fit  but  showed  no  statistically  significant  difference  between  the 
groups.  Taken  as  a  whole,  these  tests  indicate  that  there  is  likely  to  be  no  meaningful  difference 
in  the  amount  of  follow-up  assistance  provided  to  each  group. 


Table  13:  Distribution  of  Minutes  of  Follow-up  Assistance  Provided 
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Minutes  of  Follow-Up  Assistance 

Findings  Regarding  Relationships  Between  Nurse  Characteristics  and  Training  Needs 

The  relationships  between  the  characteristics  of  the  nurses  and  the  amount  of  subsequent 
assistance  needed  after  the  initial  training  was  explored  through  simple  bivariate  correlations  as 
well  as  through  regression  analysis.  Table  14  shows  the  correlations  between  the  number  of 
minutes  of  follow-up  training  required  and  the  available  nurse  characteristics.  The  correlation 
between  age  and  the  number  of  minutes  of  follow-up  is  statistically  significant  and  positive  -  on 
average,  as  age  increases  so  does  the  amount  of  follow-up  provided.  Two  other  variables  -  the 
scale  measuring  self-reported  computer  skills  and  the  indicator  variable  for  daily  computer  use  in 
any  setting  -  show  statistically-significant  negative  correlations.  In  other  words,  as  self-reported 
skill  levels  increase,  the  number  of  minutes  of  follow-up  assistance  decreases;  and  nurses  who 
reported  daily  use  of  a  computer  prior  to  the  implementation  of  the  EHR  system  required  fewer 
minutes  of  follow-up  assistance  than  those  who  did  not.  Considered  in  isolation,  several  other 
potential  predictors  -  educational  level  (holding  a  masters  degree),  prior  EHR  experience,  and 
use  of  computers  at  work  or  at  home  were  not  significantly  correlated  with  the  amount  of  follow¬ 
up  needed. 
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TABLE  14:  Correlations  -  Nurse  Characteristics  and  Minutes  of  Follow-up 


Use 

Self-rated 

Masters 

Previous  EHR 

Use 

computeiyhom 

Use  computer 

computer  skills 

Minutes  of 

Female 

Age 

or  higher 

Train 

EHR  Exp 

com  puter/work 

e 

daily 

(1-5) 

follow-up 

Female 

Pearson  Correlation 

1 

.065 

.068 

-.006 

-.032 

-.029 

-.070 

-.148* 

-.100 

.129* 

Sig.  (2-tailed) 

.277 

.256 

.923 

.624 

.662 

.293 

.025 

.131 

.031 

N 

281 

281 

281 

231 

231 

231 

231 

231 

230 

281 

Age 

Pearson  Correlation 

.065 

1 

-.005 

-.230** 

-.230** 

-.008 

-.313** 

-.276** 

-.396“ 

.135* 

Sig.  (2-tailed) 

.277 

.931 

.000 

.000 

.908 

.000 

.000 

.000 

.024 

N 

281 

281 

281 

231 

231 

231 

231 

231 

230 

281 

Masters  or  higher 

Pearson  Correlation 

.068 

-.005 

1 

-.057 

-.065 

.013 

.051 

.147* 

.125 

-.054 

Sig.  (2-tailed) 

.256 

.931 

.389 

.326 

.848 

.438 

.025 

.059 

.369 

N 

281 

281 

281 

231 

231 

231 

231 

231 

230 

281 

Previous  EHR  Train 

Pearson  Correlation 

-.006 

-.230** 

-.057 

1 

.882** 

.139* 

.169** 

.159* 

.290** 

-.059 

Sig.  (2-tailed) 

.923 

.000 

.389 

.000 

.033 

.009 

.014 

.000 

.362 

N 

231 

231 

231 

237 

237 

237 

237 

237 

236 

237 

EHR  Exp 

Pearson  Correlation 

-.032 

-.230** 

-.065 

.882** 

1 

.178** 

.137* 

.158* 

.274** 

-.054 

Sig.  (2-tailed) 

.624 

.000 

.326 

.000 

.006 

.035 

.015 

.000 

.405 

N 

231 

231 

231 

237 

237 

237 

237 

237 

236 

237 

Use  computer/work 

Pearson  Correlation 

-.029 

-.008 

.013 

.139* 

.178** 

1 

.158* 

.066 

.127 

-.084 

Sig.  (2-tailed) 

.662 

.908 

.848 

.033 

.006 

.015 

.312 

.051 

.195 

N 

231 

231 

231 

237 

237 

237 

237 

237 

236 

237 

Use  computer/home 

Pearson  Correlation 

-.070 

-.313** 

.051 

.169** 

.137* 

.158* 

1 

.243** 

.488** 

-.016 

Sig.  (2-tailed) 

.293 

.000 

.438 

.009 

.035 

.015 

.000 

.000 

.811 

N 

231 

231 

231 

237 

237 

237 

237 

237 

236 

237 

Use  computerdaiy 

Pearson  Correlation 

-.148* 

-.276** 

.147* 

.159* 

.158* 

.066 

.243** 

1 

.433** 

-.212** 

Sig.  (2-tailed) 

.025 

.000 

.025 

.014 

.015 

.312 

.000 

.000 

.001 

N 

231 

231 

231 

237 

237 

237 

237 

237 

236 

237 

Self-rated  computer  skills 

Pearson  Correlation 

-.100 

-.396** 

.125 

.290** 

.274** 

.127 

.488** 

.433** 

1 

-.175** 

(1-5) 

Sig.  (2-tailed) 

.131 

.000 

.059 

.000 

.000 

.051 

.000 

.000 

.007 

N 

230 

230 

230 

236 

236 

236 

236 

236 

236 

236 

Minutes  of  folbw-up 

Pearson  Correlation 

.129* 

.135* 

-.054 

-.059 

-.054 

-.084 

-.016 

-.212** 

-.175** 

1 

Sig.  (2-tailed) 

.031 

.024 

.369 

.362 

.405 

.195 

.811 

.001 

.007 

N 

281 

281 

281 

237 

237 

237 

237 

237 

236 

289 

*.  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
**.  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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A  set  of  negative-binomial  regression  models  was  employed  to  further  explore  these 
relationships.  Table  15  provides  descriptive  information  for  the  regression  variables,  while  Table 
16  shows  the  results.  As  discussed  in  the  previous  section,  the  distribution  of  the  variable 
containing  the  number  of  minutes  of  follow-up  assistance  provided  to  each  participant  is  not 
normally  distributed,  but  is  skewed  to  the  right  and  is  characterized  by  a  large  percentage  of 
cases  with  values  of  zero  (as  many  nurses  did  not  require  additional  training  beyond  the  initial 
sessions).  For  this  type  of  dependent  variable,  Poisson  or  negative-binomial  models  should  be 
more  appropriate  than  the  OLS  regression  models  that  are  more  commonly  employed,  as  a  key 
assumption  for  OLS  regression  is  that  the  dependent  variable  should  be  normally  distributed 
(Berk  &  MacDonald,  2007;  McElduff  et  al,  2010)  .  In  this  case,  since  the  variance  in  the 
dependent  variable  is  greater  than  the  mean,  the  negative  binomial  models  should  provide  a 
better  fit  than  the  Poisson  models.  This  was  supported  by  the  diagnostic  statistics  for  each  model, 
which  showed  a  better  fit  for  negative  binomial  models. 

An  issue  with  the  selection  of  predictors  for  these  models  is  the  potential  problem  with 
multicollinearity  among  some  of  the  available  characteristics.  For  example,  age,  years  of 
experience,  and  years  of  employment  at  the  hospital  are  all  related.  Several  of  the  available 
measures  of  computer  skills  (daily  computer  use,  self-reported  computer  skills,  and  previous 
EHR  experience)  are  also  fairly  strongly  correlated.  Therefore,  these  models  use  only  one  of  the 
variables  from  each  of  these  areas. 

The  basic  regression  model  can  be  expressed  as  follows: 

TRAIN  =  c+  a  GENDER  +  b  AGE+  d  EDUC  +  /EHR  +  g  COMP  +  e 
Where  TRAIN  is  the  total  number  of  minutes  of  follow-up  assistance  provided  to  an  individual 
nurse;  GENDER  is  an  indicator  variable  for  female;  AGE  is  the  age  in  years  of  the  nurse;  EDUC 
is  an  indicator  variable  for  holding  a  Masters  degree  or  higher;  EHR  is  an  indicator  variable  for 
whether  the  nurse  has  previous  experience  with  EHR  systems;  and  COMP  is  an  indicator 
variable  for  whether  or  not  the  nurse  reported  using  a  computer  on  a  daily  basis.  In  a  variation  of 
this  model,  a  variable  consisting  of  self-rated  general  computer  proficiency  on  a  scale  of  1-5  was 
substituted  for  the  COMP  variable.  The  letter  c  is  a  constant  term  to  be  estimated  by  regression, 
and  e  is  a  random  error  term.  The  letters  a,  b,  d,f  and  g  are  coefficients  to  be  estimated. 


Table  15:  Summary  of  Regression  Variables 


Variable 

N 

Minimu 

m 

Maximu 

m 

Mean 

Std. 

Deviation 

Minutes  of  Follow¬ 
up  Assistance 
Provided 

289 

0 

550 

53.07 

95.37 

Gender 

(l=Female) 

281 

0 

1 

.93 

.26 

Age  (Years) 

281 

23 

76 

48.79 

10.80 

Education 
(l=Masters  or 
higher) 

281 

0 

1 

.06 

.23 

EHR  Experience 
(l=Yes) 

237 

0 

1 

.47 

.50 
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Variable 

N 

Minimu 

m 

Maximu 

m 

Mean 

Std. 

Deviation 

Use  Computer  Daily 
(l=Yes) 

237 

0 

1 

.51 

.50 

Self-rated  computer 
proficiency  (scale  of 
1-5) 

236 

1 

5 

3.28 

1.19 

Valid  N  (Listwise) 

230 

Table  16:  Regression  Results  -  Dependent  Variable:  Minutes  of  Follow-up  Assistance 


Independent 

Variable 

Coefficient 
(Standard  Error) 

Model  1 

Model  2 

Model  3 

Model  4 

Gender 

1  739*** 

1  976*** 
(.2858) 

1  73 1*** 

1  925*** 

(l=Female) 

(.3973) 

(.2929) 

(.2874) 

Age  (Years) 

.016** 

(.0052) 

.020*** 

(.0059) 

.013** 

(.0062) 

.009 

(.0064) 

Education 
(l=Masters 
or  higher) 

-  571 *** 
(.2620) 

839*** 

(.3117) 

-.698** 

(.3138) 

-  840*** 
(.3116) 

EHR 

Experience 

(l=Yes) 

- 

-.267* 

(.1385) 

-.219 

(.1397) 

-.080 

(.1445) 

Use 

Computer 

Daily 

(l=Yes) 

- 

— 

-  567*** 
(1426) 

— 

Proficiency 
scale  (1-5) 

- 

— 

- 

-  269*** 
(.0669) 

Number  of 

Cases 

281 

231 

231 

230 

Log 

Likelihood 

-1374.782 

-1131.903 

-1124.035 

-1115.714 

*Denotes  1%  significance  level;  **  denotes  5%  significance  level;  *  denotes  10%  significance  level. 


The  results  from  negative-binomial  models  are  somewhat  more  difficult  to  interpret  than  those 
from  OLS  regressions  because  the  coefficients  on  each  predictor  represent  rate  ratios,  which 
estimate  the  change  in  the  relative  (rather  than  absolute)  mean  number  of  events  between  the 
groups  rather  than  the  difference  in  means  between  the  groups.  However,  the  signs  on  the 
significant  coefficients  indicate  whether  each  predictor  was  associated  with  a  positive  or 
negative  increase  in  the  number  of  minutes  of  follow-up  required  by  each  nurse. 
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In  all  of  the  models,  being  female  was  associated  with  higher  levels  of  follow-up,  as  the  indicator 
variable  for  female  is  significant  and  positive  in  each  model.  It  should  be  noted  that  since  the 
overwhelming  majority  of  the  sample  was  female  this  may  have  influenced  this  finding.  The 
education  (Masters  or  higher)  indicator  is  also  significant  in  all  of  the  models  and  is  negative,  so 
it  is  associated  with  lower  levels  of  follow-up  training.  Prior  EHR  experience  is  significant  and 
negative  (indicating  lower  training  needs)  until  controls  are  added  for  prior  level  of  computer 
skills.  Age  is  also  significant  and  positive  in  all  but  one  model,  indicating  the  need  for  increasing 
follow-up  training  with  increasing  age.  Both  of  the  measures  of  pre-training  computer  skills  - 
the  indicator  for  daily  computer  use  and  self-rated  computer  skills  -  are  significant  and  negative, 
indicating  that  initial  levels  of  general  computer  literacy  (as  opposed  to  specific  EHR 
experience)  are  associated  with  lower  levels  of  follow-up,  even  when  controlling  for  other 
factors  such  as  age  and  education  level. 

Overall,  these  results  support  our  hypothesis  that  nurses  with  different  personal  characteristics 
require  different  levels  of  subsequent  training  and  technical  support  in  order  to  achieve  EHR 
proficiency,  and  that  nurses  with  higher  levels  of  education  and  general  computer  proficiency 
will  require  fewer  total  hours  of  training  and  support.  On  a  practical  level,  the  most  consistent 
predictors  studied  that  could  be  used  to  guide  the  design  of  training  programs  seem  to  be  daily 
computer  use  and  self-reported  computer  skill  level.  Both  of  these  predictors  were  statistically 
significant  and  negative  (indicating  less  follow-up)  in  the  bivariate  correlations  and  in  the 
multivariate  models,  suggesting  that  either  could  be  used  on  its  own  to  help  identify  users  who 
are  likely  to  need  less  intense  training. 

Interviews  with  Nurse  Educators  and  “Super  Users  Given  the  dearth  of  practical  information 
available  regarding  the  design  and  implementation  of  EHR  training,  the  researchers  conducted  a 
series  of  infonnal  interviews  with  nurse  training  and  technical  staff  to  explore  the  issues 
encountered  during  the  training  process.  The  findings  of  this  effort  are  meant  to  complement  and 
inform  the  quantitative  analysis  that  is  the  focus  of  most  of  our  scope  of  work.  The  key  issues 
identified  during  these  interviews  are  as  follows: 

•  Different  initial  skill  levels :  Nurses  arrived  at  the  trainings  with  widely  varying  levels  of 
basic  computer  proficiency.  While  many  participants  used  computers  on  a  daily  basis  and 
were  comfortable  with  basic  computing  concepts,  some  were  unfamiliar  with  even  basic 
actions  such  as  how  to  use  a  mouse  or  to  navigate  through  menu  choices.  Several  trainers 
cited  this  as  one  of  the  biggest  challenges  during  the  training,  as  the  lower-skilled 
participants  required  a  lot  of  extra  attention.  This  slowed  down  the  pace  of  the  classes, 
leading  to  frustration,  boredom,  and  disengagement  among  the  higher-skilled 
participants. 

•  Difficulties  in  scheduling  training  sessions :  Designing  a  training  schedule  that  allowed 
for  all  nurses  to  be  trained  in  a  short  period  of  time  while  maintaining  required  coverage 
in  each  unit  proved  to  be  challenging,  as  the  trainers  had  to  account  for  vacations,  sick 
days,  staff  turnover,  etc. 

•  Effectiveness  of  different  training  approaches  for  different  types  of  trainees :  Several 
trainers  expressed  frustration  regarding  the  web-based  component  of  the  training.  Some 
felt  that  it  was  ineffective  for  lower-skill  trainees,  since  the  trainers  ended  up  having  to 
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repeat  or  clarify  much  of  the  information  for  these  participants  during  the  instructor-led 
session.  However,  trainers  felt  that  for  some  higher-skilled  participants,  the  web-based 
training  would  provide  a  good  introduction  and  potentially  reduce  the  need  for  some  of 
the  classroom-based  sessions  for  this  group. 

•  Technical  issues  during  roll-out  affected  follow-up  training  efforts :  Since  the  nurses’  first 
usage  of  the  new  EHR  system  began  hospital-wide  on  the  same  day  that  the  system  was 
implemented,  trainers  noted  the  initial  difficulties  in  providing  follow-up  training  while 
technical  problems  with  the  system  were  being  resolved.  Some  elements  of  the  system 
did  not  work  smoothly  as  designed,  leading  nurses  to  become  frustrated  with  trainers  who 
could  not  resolve  technical  issues.  This  became  less  of  a  problem  is  the  successive  weeks 
as  the  system  began  to  work  as  intended. 

•  Apprehension  about  EHR  systems:  Some  nurses  expressed  apprehension  about  and 
resistance  to  the  new  system  because  they  felt  it  would  be  used  to  monitor  them  more 
closely.  Some  also  disliked  the  immediacy  with  which  information  must  be  entered  into 
the  system,  as  they  were  accustomed  to  being  able  to  “catch  up”  on  paperwork  later  in 
their  shift. 

•  Changes  to  workflow  and  other  policies  beyond  record-keeping:  Trainers  noted  during 
the  initial  phase  of  EHR  implementation  that  the  use  of  the  system  had  a  cascading  effect 
on  other  policies  and  procedures  not  directly  related  to  the  use  of  the  system.  This  in  turn 
required  new  training  on  the  changes  to  work  flow  and  procedures  generated  by  the 
system. 

The  training  personnel  made  the  following  suggestions  regarding  the  design  and  implementation 
of  EHR  training  programs  in  other  hospitals  making  the  transition  from  paper-based  systems: 

•  Sort  trainees  by  initial  levels  of  general  computer  proficiency:  Placing  nurses  with 
similar  levels  of  computer  proficiency  in  training  sessions  should  allow  for  appropriately¬ 
paced  classes  to  improve  participant  engagement. 

•  Train  in  smaller  groups  for  lower-skilled  participants:  The  standard  session  size  of 
twenty  was  viewed  by  many  trainers  too  large  for  all  but  the  most  skilled  participants. 
Some  trainers  recommended  that  class  sizes  for  trainees  with  lower  levels  of  computer 
skills  should  be  as  low  as  ten  participants. 

•  Utilize  training  for  higher-skilled  participants:  The  trainers  felt  that  nurses  with  the 
highest  level  of  existing  computer  skills  would  be  the  best  candidates  for  web-based 
training,  and  that  the  best  design  would  be  a  self-directed  introductory  module  covering 
basic  system  usage,  followed  by  instructor-led  sessions. 

•  Provide  training  close  to  the  EHR  roll-out  date:  Due  to  scheduling  requirements,  the  first 
groups  of  nurses  were  trained  up  to  two  months  before  the  “go  live”  date  for  the  system. 
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Some  trainers  identified  this  gap  as  negatively  affecting  learning  retention  and  leading  to 
additional  needs  for  follow-up. 

•  Changes  to  structure  of  training  sessions :  While  the  16-hour  training  length  was 
generally  viewed  as  adequate  for  most  nurses,  some  trainers  suggested  breaking  up  the  8- 
hour  sessions  into  4-hour  sessions  over  more  days  to  facilitate  participant  engagement. 

•  Provide  opportunities  for  practicing  with  the  EHR  system'.  Nurses  were  provided  with 
limited  opportunities  to  practice  on  a  simulated  version  of  the  system;  one  trainer 
suggested  that  greater  opportunities  for  self-directed  practice  with  the  system  be 
provided. 

•  Team  teaching  for  class  sessions :  Most  of  the  trainers  felt  that  the  twenty-person  class 
size  was  too  large  given  some  participants’  needs  for  frequent  individual  assistance.  The 
use  of  multiple  instructors  or  assistants  to  provide  one-on-one  help  during  the  sessions 
should  allow  the  primary  instructor  to  move  through  the  training  material  more 
efficiently. 

KEY  RESEARCH  ACCOMPLISHMENTS:  The  following  are  the  key  findings  of  our  study: 

•  Our  study  found  that  the  substitution  of  a  web  -based  component  for  a  portion  of  an 
initial  EHR  training  course  produced  no  measureable  impact  on  training  outcomes  as 
compared  to  a  fully  classroom-based  approach.  There  were  no  statistically-significant 
differences  between  the  two  training  groups  on  two  post-training  tests  of  system 
knowledge,  on  supervisor-rated  proficiency  scores  with  the  system  following 
implementation,  or  on  the  amount  of  time  needed  for  follow-up  assistance.  These 
findings  did  not  support  our  hypothesis  that  nurses  who  receive  web-based  training  as  a 
substitute  for  part  of  the  classroom-based  training  will  show  higher  initial  levels  of 
knowledge  of  EHR  system  procedures.  However,  it  should  be  noted  that  this  change  in 
the  training  model  produced  essentially  the  same  results  as  the  fully  classroom-based 
model,  which  may  have  implications  regarding  reducing  overall  training  costs  as 
discussed  in  the  Conclusions  section  below. 

•  On  average,  nurses  needed  about  53  minutes  of  follow-up  training  and  assistance  to 
achieve  proficiency  with  the  system.  However,  the  amount  of  such  training  varied 
widely,  with  half  of  the  nurses  needing  no  additional  training  but  others  requiring  as 
much  as  9  hours. 

•  Our  study  found  evidence  to  support  our  hypothesis  that  nurses  with  different  personal 
characteristics  require  different  levels  of  subsequent  training  and  technical  support  in 
order  to  achieve  EHR  proficiency,  and  that  nurses  with  higher  levels  of  education  and 
general  computer  proficiency  will  require  fewer  total  hours  of  training  and  support. 
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•  Two  measures  of  pre-training  computer  skill  levels  -  an  indicator  for  daily  computer  use 
and  a  higher  self-rating  of  computer  skills  -  were  consistently  associated  with  less 
follow-up  assistance,  even  when  controlling  for  other  factors  such  as  age  and  education 
level.  These  may  therefore  be  the  most  useful  among  the  set  of  predictors  tested  for 
screening  nurses  for  assignment  into  training  courses  of  different  levels  of  intensity. 

•  Increasing  age  was  associated  with  higher  levels  of  need  for  follow-up  assistance  in  most 
models. 

REPORTABLE  OUTCOMES:  At  this  point,  our  project  does  not  have  any  of  the  listed 
reportable  outcomes  as  we  have  just  completed  the  final  data  analysis.  We  anticipate  submitting 
an  article  for  publication  based  on  the  results  to  an  appropriate  journal  in  the  near  future. 

In  the  meantime,  we  submitted  brief  descriptions  of  our  project  and  the  results  to  MD  News  and 
the  Local  Edge  which  reach  a  variety  of  hospital  personnel,  physicians  and  others. 
Communicating  through  these  outlets  allows  us  to  reach  practicing  professionals  in  positions  to 
introduce  our  findings  into  their  own  organizations  as  they  train  staff  on  HIT,  and  specifically, 
EHR  systems. 

CONCLUSION:  The  findings  of  this  study  suggest  a  potential  path  to  reducing  the  cost  of 
training  programs  related  to  the  roll-out  of  EHR  systems  within  military  (as  well  as  civilian) 
hospitals.  Our  first  key  finding  is  that  the  substitution  of  a  web-based  component  for  part  of  the 
training  curriculum  as  opposed  to  only  utilizing  standard  classroom  training  had  no  measureable 
effect  on  training  outcomes.  The  groups  showed  no  statistically-significant  differences  on  three 
different  measures  of  initial  knowledge/proficiency  or  in  the  number  of  minutes  of  additional 
assistance  required  after  the  initial  training  course.  These  findings  did  not  support  our  hypothesis 
that  nurses  who  receive  web-based  training  as  a  substitute  for  part  of  the  classroom-based 
training  will  show  higher  initial  levels  of  knowledge  of  EHR  system  procedures.  However,  these 
results  do  imply  that  some  part  of  the  training  could  be  delivered  using  web-based  methods 
without  affecting  implementation  outcomes,  which  should  be  less  expensive  than  the  delivery  of 
classroom-based  training.  This  may  be  particularly  true  for  certain  types  of  users,  especially 
those  who  have  higher  levels  of  general  computer  proficiency  prior  to  the  training. 

Though  we  did  not  measure  the  speed  at  which  trainees  completed  the  web-based  component  of 
the  course,  reports  from  the  instructors  indicated  that  some  computer-proficient  trainees 
completed  these  modules  much  more  quickly  than  other  participants.  Since  much  of  the  expense 
of  EHR  training  for  nurses  is  related  to  the  cost  of  paying  trainees’  hourly  wages  during  training 
(which  is  likely  to  be  at  overtime  rates  since  unit-level  coverage  must  be  maintained  within  the 
facility  during  the  training),  any  approach  that  reduces  the  number  of  hours  spent  in  training 
sessions  should  substantially  reduce  costs  when  multiplied  across  hundreds  or  thousands  of 
employees.  The  added  flexibility  in  scheduling  training  sessions  provided  by  the  web-based 
training  may  also  generate  cost  savings  by  reducing  the  number  of  “backfill”  hours  required  (in 
which  another  nurse  must  cover  for  a  nurse  in  training  to  ensure  sufficient  coverage  within  the 
unit).  Since  the  web-based  training  hours  could  be  spread  across  shorter  sessions  on  multiple 
days,  this  should  reduce  the  need  for  such  “backfill”  scheduling  during  the  training  period. 
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Our  second  key  finding  -  that  nurses’  characteristics  are  related  to  the  amount  of  follow-up 
training  needed  -  may  also  be  used  to  guide  the  development  and  implementation  of  training  and 
technical  assistance  programs.  In  our  case,  the  hospital’s  training  staff  were  largely  left  to  their 
own  devices  in  the  design  of  the  initial  training  program  by  the  EHR  vendor,  whose  staff  was 
focused  mainly  on  technical  issues  with  the  system  as  well  as  the  provision  of  roving  trainers 
during  the  post-“go  live”  period  of  system  implementation.  Since  the  switchover  to  an  EHR 
system  from  a  paper-based  system  is  generally  a  unique  event  within  a  hospital,  it  is  likely  that 
the  personnel  charged  with  designing  and  delivering  the  initial  on-site  training  will  have  limited 
knowledge  of  what  to  expect.  Our  findings  suggest  that  rather  than  a  “one  size  fits  all”  approach 
to  the  initial  training,  hospitals  should  vary  the  length  and  type  of  initial  training  based  on  their 
general  level  of  computer  proficiency.  This  factor  appears  to  be  a  better  predictor  of  success  in 
achieving  competency  with  the  EHR  system  than  other  factors  tested,  including  prior  EHR 
experience  and  educational  attainment.  Smaller  class  sizes  with  multiple  instructors  for  nurses 
with  low  levels  of  initial  computer  proficiency  may  also  improve  training  outcomes  and  reduce 
the  need  for  extensive  follow-up  assistance.  Overall,  targeted  training  programs  based  on  initial 
screening  for  these  factors  could  generate  substantial  cost  savings  in  large  hospital  systems  such 
as  military  hospitals  as  the  nationwide  transition  and  upgrades  to  EHR  systems  continues  over 
the  next  few  years.  This  approach  could  also  continue  to  produce  cost  savings  following  EHR 
upgrades  that  require  additional  training. 
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Sound  Shore  Medical  Center 

Health  Technology  Integration  for  Clinical,  Patient  Records  and  Financial  Management 

Related  to  the  Military 


DATA  FIELDS  TO  BE  COLLECTED  FROM  EMPLOYMENT  RECORDS  REGARDING 

STUDY  PARTICIPANTS 

•  AGE 

•  GENDER 

•  JOB  TITLE 


o  Registered  Nurse 
o  Clinical  Nurse  1 
o  Clinical  Nurse  2 

o  Clinical  Nurse  3/Assistant  Nursing  Care  Coordinator 
o  Assistant  Administrative  Nurse  Coordinator 


•  NUMBER  OF  YEARS  EMPLOYED  BY  SSMC/MVH 

•  NUMBER  OF  YEARS  OF  NURSING  EXPERIENCE 

•  EDUCATIONAL  LEVEL  (HIGHEST  DEGREE  COMPLETED) 

o  Diploma 
o  Associates  Degree 
o  B.S.N.  or  B.S. 
o  M.S.N.  or  M.S. 

•  CERTIFICATION/LICENSE  TYPE 

o  R.N. 
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INITIAL  SURVEY  RE:  COMPUTER  LITERACY/EXPERIENCE 

Thank  you  for  taking  this  survey.  This  survey  is  part  of  a  federally-funded  research  project  which  will 
measure  the  effectiveness  of  the  Electronic  Health  Records  system  training  and  find  ways  to  improve  it  in 
the  future.  Participation  in  the  survey  is  voluntary.  The  information  you  provide  will  only  be  used  in  the 
evaluation  of  the  effectiveness  of  the  Electronic  Health  Record  (EHR)  training  program.  Your  answers  to 
these  questions  will  not  be  used  as  part  of  employee  performance  evaluations  and  will  not  be  provided  to 
your  immediate  supervisor (s).  The  data  provided  to  the  researchers  will  not  include  any  information 
which  could  be  used  to  identify  you. 

The  researchers  are  Social  Policy  Innovations,  Inc.  For  questions  regarding  this  research,  please  e-mail 
tbs 203(a),. smail.com. 

Your  Identification  Code _ 

Please  circle  your  answer  for  each  of  the  following  questions. 

1.  Have  you  ever  received  training  in  the  use  of  an  Electronic  Health  Records  system? 

Yes  No 

2.  Have  you  ever  used  an  Electronic  Health  Records  system  to  record  information  regarding  patient 
care? 

Yes  No 

3.  If  yes,  for  how  long  did  you  use  the  Electronic  Health  Records  system? 

A.  Less  than  six  months 

B.  Six  months  to  1  Year 

C.  1-2  years 

D.  3-4  Years 

E.  More  than  4  Years 

4.  Do  you  currently  use  computers  as  part  of  your  work  in  the  hospital? 

Yes  No 

5.  On  a  scale  of  1  to  5,  (with  1  being  not  sure  how  to  turn  a  computer  on,  and  5  being  very  good), 
how  would  you  rate  your  computer  skills? 

1  2  3  4  5 

4.  Do  you  use  a  computer  at  home? 

5. 

Yes  No 

6.  Do  you  have  access  to  the  internet  at  home? 

Yes  No 
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See  next  page  for  last  question. 

7.  How  often  do  you  use  computers,  including  at  work  and  at  home  (please  choose  the 
answer  that  is  closest  to  the  frequency  of  your  computer  usage)? 

A.  Never 

B.  Once  per  month 

C.  Once  per  week 

D.  Several  times  per  week 

E.  Daily 
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Employee  EHR  Training  Survey 


Thank  you  for  taking  this  survey.  All  responses  will  be  anonymous  and  no 
attempt  will  be  made  to  identify  survey  respondents.  This  survey  is  part  of  a 
federally- funded  research  project  which  seeks  to  determine  the  relative 
effectiveness  of  different  types  of  training  in  the  use  of  electronic  health 
records  systems.  The  survey  will  give  the  trainers  and  researchers  a  better 
idea  about  how  successful  the  electronic  medical  records  training  was  and  ways 
to  improve  it  in  the  future.  Participation  in  the  survey  is  voluntary,  and  the 
information  you  provide  will  only  be  used  in  the  evaluation  of  the 
effectiveness  of  the  electronic  medical  records  training  program.  Your  answers 
to  these  questions  will  not  be  used  as  part  of  employee  performance 
evaluations  and  will  not  be  provided  to  your  immediate  supervisor (s) .  The  data 
provided  to  the  researchers  will  not  include  any  information  which  could  be 
used  to  identify  you.  The  researchers  are  Social  Policy  Innovations,  Inc. 

For  questions  regarding  this  research,  please  e-mail  tbs203@gmail.com 

Basic  information  (circle  or  fill-in) 


Hospital:  Sound  Shore  Medical  Center  Mt .  Vernon  Hospital 

Age:  18-24  25-34  35-44  45-54  55-64  65-74 

Gender:  Male  Female 

Education  Level:  B.A./B.S.  Graduate/Professional 

Years  licensed  in  the  medical  profession:  0-2  3-5  6-8  9+ 

Native  language:  English  Other 

In  your  opinion,  how  good  are  your  computer  skills? 

Very  high  High  Average  Low  Very  Low 

Have  you  ever  used  an  electronic  health  records  system  in  another  job? 

Yes  No 

If  Yes,  for  how  many  years?  _ 

Have  you  ever  received  training  in  the  use  of  an  electronic  health  records 
system  in  another  job? 

Yes  No 

Do  you  use  a  computer  at  home?  Yes  No 

About  the  electronic  health  records  training  at  SSHS 

How  long  was  your  training  (hours)? _ #  hours 

Did  part  of  your  training  involve  web-based  training? 

Yes  No 

SEE  NEXT  PAGE 


How  much  did  you  know  about  the  electronic  health  records  system  before  the 
training? 
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a  lot  a  little  nothing 

How  comfortable  are  you  with  the  new  system  after  the  training? 

very  comfortable  somewhat  comfortable  not  at  all  comfortable 


Please  rate  how  you  feel  about  the  following  statements  (circle  a  number) ; 
The  old  (paper  records)  system  is  more  efficient. 

Definitely  Not  Sure  Definitely  Not 

5  4  3  2  1 


The  new  Electronic  Health  Record  system  will  result  in  better  patient 
outcomes . 

Definitely  Not  Sure  Definitely  Not 

5  4  3  2  1 


The  new  system  will  make  my  work  easier. 

Definitely  Not  Sure  Definitely  Not 

5  4  3  2  1 


I  received  adequate  training  on  the  new  system. 

Definitely  Not  Sure  Definitely  Not 

5  4  3  2  1 


My  manager  visibly  commits  to  the  implementation  of  the  new  Electronic  Health 
Record  system. 

Definitely  Not  Sure  Definitely  Not 

5  4  3  2  1 


Information  will  be  more  completely  recorded  in  the  new  system. 

Definitely  Not  Sure  Definitely  Not 

5  4  3  2  1 


Information  will  be  more  accurately  recorded  in  the  new  system. 

Definitely  Not  Sure  Definitely  Not 

5  4  3  2  1 

I  feel  ready  to  use  the  new  Electronic  Health  Record  System. 

Yes _  No _ 

SEE  NEXT  PAGE 
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Other  questions 

What  is  the  most  useful  thing  you  learned  in  the  training? 

How  could  the  training  be  improved  to  enable  you  to  learn  more  about  the 
electronic  medical  record  system? 

What  else,  if  anything,  would  you  have  liked  to  learn  about?  Please  give  clear 
examples . 


Nursing  Computer  Competency  Post  Test 
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This  test  is  part  of  our  overall  effort  to  ensure  that  nurses  achieve  competency  on  the  new  Electronic  Health  Record 
(EHR).  Your  answers  to  these  questions  will  not  be  used  as  part  of  employee  performance  evaluations  and  will  not 
be  provided  to  your  immediate  supervisor (s). 

We  are  also  participating  in  a  federally-funded  research  project  which  will  measure  the  effectiveness  of  the  EHR 
training  and  find  ways  to  improve  it  in  the  future.  Your  participation  in  the  research  project  is  voluntary’.  Your 
answers  will  be  provided  to  researchers  who  will  not  have  any  information  which  could  be  used  to  identify  you.  The 
information  provided  to  the  researchers  will  only  be  used  in  evaluating  the  effectiveness  of  the  EHR  training 
program. 

Your  Identification  Code 


Instructions:  Please  circle  the  one  best  answer  for  each  question  below.  Each  question  is 
worth  2  points 

1.  Which  of  the  following  statements  regarding  SUNRISE  CLINICAL  MANAGER  (SCM) 
security  are  true? 

A.  Your  assigned  security  gives  you  access  to  carry  out  functions  in  SCM. 

B.  SCM  records  all  of  your  actions.  Every  time  you  access  a  different  chart  section 
or  whenever  you  make  changes  to  the  patient  data,  SCM  logs  that  information 
and  creates  an  audit  trail. 

C.  Every  time  you  leave  the  computer,  it  is  very  important  that  you  log  off  SCM 
otherwise,  the  next  person  to  use  SCM  on  that  computer  will  be  entering  data 
under  your  name,  and  it  will  be  logged  in  the  audit  trail  as  such. 

D.  All  of  the  above  statements  are  true. 

2.  The  Auto  Log-Off  Feature  addresses: 

A.  Infection  control 

B.  Security 

C.  Power  usage 

D.  Physician  liability 

3.  What  information  about  a  patient  can  be  found  in  the  Worklist  Manager? 

A.  Scheduled  procedures,  departmental  tasks,  and  the  MAR 

B.  Patient  location,  demographics,  and  height/weight 

C.  Allergies  and  Alerts 

D.  Results  and  Health  Issues 

4.  The  three  types  of  patient  lists  you  can  create  are: 

A.  Criteria  based,  Temporary,  Special. 

B.  Role  based,  Permanent,  Location. 

C.  Orders,  Results,  MAR 

D.  Provider,  location,  Allergy 

5.  The  list  of  patients  in  a  specific  location  is  an  example  of  a: 

A.  Special  list 

B.  Permanent  list 

C.  Temporary  list 

D.  Criteria-based  list 
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6.  What  does  the  following  icon  in  the  Orders  tab  indicate? 

A.  All  orders  displayed  are  Active 

B.  Orders  are  verified  by  the  Pharmacy 

C.  The  view  has  been  filtered 

D.  All  orders  are  displayed 


7.  To  document  that  you  have  collected  a  urine  sample  for  a  culture,  which  action  button  in  the  Orders  tab 
must  be  selected? 

A.  Approve/Verify... 

B.  Add  Specimen... 

C.  Release... 

D.  DC/Cancel... 


8.  The  Clinical  Summary  tab  provides  a  view  of  which  commonly  used  nursing 
document? 

A.  Flow  Sheet 

B.  Nursing  Kardex 

C.  SOAP  note 

D.  MAR 

9.  What  information  must  be  entered  and  marked  as  reviewed  prior  to  entering  an  order? 

A.  Name 

B.  Birth  date 

C.  Allergies 

D.  Weight 

10.  In  the  Patient  Info  tab,  you  can  edit  what  type  of  patient  data? 

A.  Allergies 

B.  Demographics 

C.  Health  Issues 

D.  Orders 

1 1 .  Which  of  the  following  worklists  can  be  viewed  to  find  out  when  a  patient’s  CT  scan  has  been 

scheduled? 

A.  Procedures-  Consolidated 

B.  Department  Tasks-Consolidated 

C.  MAR-  This  Pt  by  Order  Set 

D.  Nursing  Tasks  by  Patient  List 

12.  When  entering  a  telephone  order  from  a  physician,  which  button  on  the  Order  Entry 

Worksheet  must  be  selected? 

A.  Requested  by:  me 

B.  Requested  by:  other 

C.  Patient  allergy  review 

D.  Start  of  browse 

13.  A  green  flag  will  display  in  the  Check  Orders  or  New  Orders  column  when: 

A.  A  routine  order  has  been  placed 

B.  A  STAT  order  has  been  placed 

C.  Results  are  available 
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D.  An  order  has  been  completed 


14.  To  delete  an  incorrect  order  in  the  Review  section  of  the  Order  Entry  Worksheet  which  button  can 
be  selected? 

A.  Edit 

B.  Delete 

C.  Copy 

D.  Mark  as  done 

15.  Right-clicking  in  the  time  blocks  on  the  MAR  will  allow  you  to: 

A.  Delete  the  task 

B.  Mark  as  done 

C.  Cancel  the  dose 

D.  View  results 


16.  The  Resuscitation/Care  orders  column 

A.  Notes  the  presence  of  specific  resuscitation  care  orders 

B.  Allows  you  to  view  resuscitation  orders 

C.  Will  display  information  such  as  “DNR/DNI” 

D.  Is  updated  by  the  Physician 


17. 


Which  of  the 


A. 

B. 

C. 

D. 


following  Icons  do  you  need  to  click  to  refresh  the  page? 


1 8.  By  using  the  Right-click  in  the  Orders  tab  or  the  Worklist  Manager  you  can  view: 

A.  Order  details  and  history 

B.  Relevant  results 

C.  Drug  interactions 

D.  Patient  Allergies 

19.  Patient  Alerts  can  be  viewed  by  double-clicking  the: 

A.  Patient’s  name 

B.  Flag  in  the  Alert  column 

C.  Flag  in  the  Check  Orders  Column 

D.  Patient  Header 

20.  How  would  you  locate  a  completed  CBC  result  in  SCM? 

A.  Ask  the  patient’s  family  members 

B.  View  the  results  under  the  Results  Tab 

C.  Call  the  lab  to  ask  about  CBC  results 
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D.  Look  at  the  bedside  flowsheet 


Part  II  -  Training  Assessment:  Nursing  Order  Entry 


This  test  is  part  of  our  overall  effort  to  ensure  that  nurses  achieve  competency  on  the  new  Electronic  Health  Record 
(EHR).  Your  answers  to  these  questions  will  not  be  used  as  part  of  employee  performance  evaluations  and  will  not 
be  provided  to  your  immediate  supervisor (s). 

We  are  also  participating  in  a  federally-funded  research  project  which  will  measure  the  effectiveness  of  the  EHR 
training  and  find  ways  to  improve  it  in  the  future.  Your  participation  in  the  research  project  is  voluntary’.  Your 
answers  will  be  provided  to  researchers  who  will  not  have  any  information  which  could  be  used  to  identify  you.  The 
information  provided  to  the  researchers  will  only  be  used  in  evaluating  the  effectiveness  of  the  EHR  training 
program. 

Your  Identification  Code 


I  nstructions:  Using  your  assigned  patient,  complete  the  activities  as  outlined.  Part  1 1 
counts  for  60  points  toward  your  final  score 

Nursing/RN  and  LPN  -  Order  Entry 

Order  Entry  Activity  1 :  (6  pts  total) 

Scenario 

Your  patient,  a  59  year  old  type  2  diabetic  client  (noninsulin-dependent)  for  6  years, 
presented  to  his  physician  office  with  a  non  healing  ulcer  of  2  weeks’  duration  on  his/her  left 
foot.  Screening  tests  done  in  the  doctor’s  office  revealed  blood  glucose  of  346/finger  stick 
and  urine.  Because  of  the  distance  from  medical  provider  and  lack  of  community  services, 
he/she  is  admitted  to  the  hospital.  (Note:  Your  patient  is  not  pregnant) 

Update  to  Patient  Information 

1 )  Create  a  manual  list  with  your  assigned  patient. 

2)  Your  patient  reported  that  he/she  is  allergic  to  bees-with  a  reaction  of  Swelling/Edema. 

3)  Your  patient  is  taking  the  following  Home  Medication:  Simvastatin  10  mg  tablet,  1 
tablet  daily. 

Score: _ 

Continued  on  next  page 
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Order  Entry  Activity  2  :  (24  pts  total) 

Douglass  Ohio,  the  admitting  physician,  requested  the  following  telephone  orders: 

1 )  Gram’s  Stain  of  foot  ulcer;  Source  =  skin;  Site  =  foot. 

2)  Blood  Glucose  finger  stick  4  times  a  day. 

3)  CBC  Next  Draw 

4)  Electrolytes,  next  draw  and  daily  for  3  occurrences. 

5)  Chest  X-ray  2  View  P  A /  Lateral-  Current  signs  and  symptoms=  chest  pain; 
requesting  physician  number  =  712345 

6)  ECG  12  Lead-  indication=  Diabetes,  requesting  physician  number  =  712345 

7)  Humulin  N  Units=  10;  route  =  Subcutaneous,  frequency=  morning  and  evening; 
begin  insulin  instruction  for  post  discharge  self-care 

8)  Darvocet  -N  100  mg  every  4  hours  prn  for  pain. 

9)  Diabetic  diet  -  2000-2200  calorie  diet — add  note  3  meals/2  snacks;  Dietitian  consult 
(nutrition  assessment,  and  education) 

1 0)  Vital  Signs  4  times  a  day. 

1 1 )  Activity  -  may  be  up  in  chair 

12) Submit  your  orders  and  authenticate 

Score: _ 

Continued  on  next  page 
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Order  Entry  Activity  3:  ( 14  pts  total) 

Your  patient  had  an  acute  Asthma  attack  during  their  hospital  visit.  Enter  the  following  order 
on  behalf  of  Douglas,  Ohio  (hint:  tapering  dose): 

Enter  the  telephone  order  for  Prednisone: 

1 )  40  mg  every  1 2  hours  x  3  days 

2)  40  mg  every  day  x  3  days 

3)  30  mg  every  day  x  2  days 

4)  20  mg  every  day  x  2  days 

5)  1 0  mg  every  day 

Orders  Tab  Review  and  Orders  Management 

1 )  Order  Glucophage  1 000  mg  PO  twice  a  day 

2)  Cancel  the  Humulin  Insulin  order. 

Score: _ 

Continued  on  next  page 
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Order  Entry  Activity  4  :  (12  pts  total) 

You  have  assessed  your  patients  foot  wound.  Document  the  following: 

a)  Site  =  Foot 

b)  Irrigate  With  =  Normal  Saline 

c)  Cover  With  =  Other 

d)  Comments/Instructions  =  Cover  with  wet  to  dry  sterile  dressing 

Your  patient  is  not  responding  to  current  wound  treatment.  The  physician  instructs  you  to: 

1 )  Place  an  order  for  a  Wound  Care  Nurse  Consult. 

2)  Use  the  Orders  Tab  to  discontinue/cancel  the  “Up  in  Chair”  order  because  alternate 
treatment  is  necessary. 

Score: _ 

Continued  on  next  page 
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Order  Entry  Activity  5  (4  pts  total) 

Scenario 

Your  patient  has  been  admitted  to  the  hospital  with  an  infectious  Disease.  A  “Notify  Infection 
Control  Nurse”  consult  order  was  placed. 

Order  Entry 

1 .  Enter  the  Airborne  Isolation  (Precautious,  Airborne)  Standard  Interventions  order.  The 
reason  for  lsolation=  Shingles,  Disseminated. 

Discontinue  Order 

2.  The  MD  has  given  you  a  telephone  order  that  the  patient’s  no  longer  needs  isolation. 
Discontinue  the  isolation  order 

Score: _ 


When  complete,  call  the  Instructor  or  TCE  to  verify  your  work. 


Score: 

Part  I: 
Part  1 1 : 
Total: 


rev.3/22/11 
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Post  Go-Live  SCM  Nursing  Assessment 


Nurse  Name: 
Nurse  HBO 
Code: 

Unit: 

Assessment 

Dt: 

Assessment 
Completed  by: 


This  test  is  part  of  our  overall  effort  to  ensure  that  nurses  achieve 
competency  on  the  new  Electronic  Health  Record  (EHR).  We  are 
also  participating  in  a  federally-funded  research  project  which  will 
measure  the  effectiveness  of  the  EHR  training  program  and  find 
ways  to  improve  it  in  the  future.  Your  participation  in  the 
research  project  is  voluntary.  Your  answers  will  be  provided  to 
researchers  who  will  not  have  any  information  which  could  be 
used  to  identify  you.  The  information  provided  to  researchers  will 
only  be  used  in  evaluating  the  effectiveness  of  the  EHR  training 
program. 


Instructions: 

Please  use  the  key  to  assess  each  competeny  statement: 
5=independent  4=minimal  assist  3=moderate  assist 

2=  maximum  assist  1=  unable  to  demonstrate 

Demonstrates  ability  to  find  a  patient  that  is  not  on  their  unit  list. 

5 

4 

3 

2 

1 

Demonstrates  ability  to  create/maintain  personal  patient  list 

5 

4 

3 

2 

1 

Demonstrates  steps  to  turn  on/off  flags 

5 

4 

3 

2 

1 

Articulates  the  purpose  of  new  [orders,  results,  alerts  and 
documents]  flag  and  demonstrate  clearing  of  the  appropriate  flag. 

5 

4 

3 

2 

1 

Articulates  the  significance  of  the  colors/shapes  (ie  red-square, 
green-triangle) 

5 

4 

3 

2 

1 

Articulates  the  purpose  of  the  to  sign  and  to  verify  flags  and  is 
able  to  demonstrate  co-signing  and  verifying  orders. 

5 

4 

3 

2 

1 

Demonstrates  where  patient  allergy  information  is 
entered/stored. 

5 

4 

3 

2 

1 

Demonstrates  how  to  complete  and  discontinue  an  order. 

5 

4 

3 

2 

1 
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Demonstrates  the  ability  to  enter  a  verbal  medication  order. 

5 

4 

3 

2 

1 

CONTINUED  ON  BACK  OF  THIS  PAGE 

Demonstrates  the  ability  to  generate  a  lab  label/collect  a 
specimen  for  a  given  laboratory  test 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  complete  an  admission  profile 
(including  the  purpose  of  the  !) 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  find  information  in  a  specific 
document  (such  as  the  H/P  or  progress  note) 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  enter  vital  signs  (including  adding 
parameters  and  time  columns) 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  discontinue  vital  signs/intake  & 

Output 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  enter  Intake  &  Output. 

5 

4 

3 

2 

1 

Articulates  the  purpose  of  shift  and  daily  totals,  length  of  stay  and 
blood  balance  totals. 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  enter  observations  on 
assessment/cares  flowsheet  (including  adding  parameters  and 
time  columns) 

5 

4 

3 

2 

1 

Articulates  the  meaning  of  WDL 

5 

4 

3 

2 

1 

Articulates  the  meaning  of  color  on  the  eMar  (both  on  the  task 
description  as  well  as  a  specific  task  instance. 

5 

4 

3 

2 

1 

Demonstrates  how  to  mark  a  medication  as  done 

5 

4 

3 

2 

1 

Articulates  when  the  "Mark  as  Not  Done"  and  "Mark  as  Done  by 
Other"  selections  should  be  used. 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  reschedule  a  single  instance  of  a  task 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  reschedule  all  instances  of  a  task 

5 

4 

3 

2 

1 

Articulates  when  "Add  a  completed  task"  function  should  be  used. 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  add  a  comment  to  an  eMar  task  and  a 
flowsheet  observation 

5 

4 

3 

2 

1 

Articulates  the  purpose  of  the  clinical  summary  tab.  Articulates 
process  for  professional  exchange  handoff. 

5 

4 

3 

2 

1 

Demonstrates  the  ability  to  complete  a  patient  discharge. 

5 

4 

3 

2 

1 
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Total  Score: _ 

SEND  COMPLETED  SURVEY  TO  SUE  McLEER  (SSMC) 
OR  J  RCANDA(MVH) 
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EHR  TRAINING/TECHNICAL  ASSISTANCE  ROSTER 

As  part  of  a  federally-funded  evaluation  of  the  nurse  training  program  for  the  new  Electronic  Health  Records  (EHR) 
system,  we  are  tracking  the  amount  of  technical  assistance  provided  to  each  nurse  after  the  completion  of  the 
classroom-based  training  sessions.  Please  record  the  following  information  each  time  you  provide  assistance  with 
the  use  of  the  new  system  to  a  nurse. 

Please  note  that  the  nurse ’s  participation  in  the  evaluation  is  voluntary.  The  information  you  provide  is  to  be  used 
only  in  the  evaluation  of  the  effectiveness  of  the  EHR  training  program.  The  data  provided  to  the  researchers  will 
not  include  information  which  could  be  used  to  identify  the  nurse  or  the  trainer. 


TRAINER: 


EMPLOYEE  NAME  - 
please  print 

EMPLOYEE 

DEPARTMENT 

EMPLOYEE  ID 
(HBO)  NUMBER 

DATE 

NUMBER  OF 
MINUTES  OF 
ASSISTANCE 
PROVIDED 

SEND  COMPLETED  FORM  TO  SUE  McLEER  (SSMC)  OR  JR  CANDA 
(MVH) 
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SOUND  SHORE  MEDICAL  CENTER  OF  WESTCHESTER 


“Health  Technology  Integration  for  Clinical,  Patient  Records  and  Financial  Management 

Related  to  the  Military” 

W81XWH-10-2-0060 

PERSONNEL 


Title 

Name 

HIT  Training  Coordinator 

Sue  McLeer 

Protocol  Developer 

Thomas  Saunders 

Assistant  Evaluator 

Thomas  Saunders 

HIT  Training  Consultant 

Cary  Steiner 
Sandra  Wagner 

Research  Implementation 

Janette  Cooke 

Data  Entry  Services 

Thomas  Saunders 
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